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240 CHAPTERS. BIT ALLOCATION AND TRANSFORM CODING 

Since there is an infinite variety of orthogonal matrices to choose from 
all of which prevent magnification of quantizing noise, a natural approach 
>s to find a matr.x T which eliminates or nearly eliminates the correlation 
between the transform coefficients. We shall see that the effect of elimi- 
nating correlation is to reduce the product of the variances of the vector 
components, which in turn will reduce the average distortion attainable for 
the given bit quota when the optimal bit allocation under the hieh rate 
approximation of (8.3.1) is used. 

8.6 Karhunen-Loeve Transform 

In general, the components of X are correlated with one another How- 
ever, it is indeed possible to select an orthogonal matrix T, for a given pdf 
describing X, that will make Y = TX have pairwise uncorrected compo- 
nents. This choice of transform matrix that makes the linear transforma- 
tion have this desirable property is called the discrete time Karhunen-Loeve 
transform or the Hotelling transform and is denned below. 

Let R x = £[XX'j. denote the autocorrelation matrix of the input Col- 
umn) vector X. (If the vectors are complex this becomes R x = E\XX'] > 
Let u, denote the eigenvectors of R x (normalized to unit norm) and A, th* 
corresponding eigenvalues. Since any autocorrelation matrix is symmetric 
and nonnegauve definite, there are k orthogonal eigenvectors and the corre- 
sponding eigenvalues are real and nonnegative. Without loss of generality 
we assume the indexing is such that 

Ai > A 2 > • •• > Ai > 0. 

The Karhunen-Loeve transform matrix is then defined as T = U f . where 

U = [uiu 2 -Ui], 

that is, the columns of U are the eigenvectors of R A -. 
Then the autocorrelation matrix of Y is given by 



Ry = E[YY'] = E'fU'XX'U] 

rA r o • 



= U'R A -U = 



0 A, 



L o 



••• 0 1 
... o 

0 A t J 



Thus we see that the Karhunen-Loeve transform does indeed decorre- 
late the mput vector. It also follows that the variances of the transform 
coefficients are the eigenvalues of the autocorrelation matrix R x 
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8.6. KARHUNEN-LOEVE TRANSFORM 
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Suppose we apply optimal quantization to each transform coefficient. 
For optimal bit allocation, the mean square distortion resulting from trans- 
form, coding with the Karhunen-Loeve transform is given by 



0tc = *ff(I[Ai)*2 



-26 



i = l 



where H is the geometric mean of the quantization coefficients hi for the 
normalized pdf's VJ. It is important to note that H depends on the trans- 
form matrix T and on the joint pdf of the input vector X. In the special 
case where X is Gaussian, each Y { is also Gaussian so that H = h g . the 
quantization coefficient for the Gaussian pdf obtained by applying (8 2 3) 
for the Gaussian case and given by (8.2.4). We have used the fact that the 
variance of the coefficient Y ( is A, . Since the determinant of a matrix is the 
product of its eigenvalues, we have 



£> tc = fc//(detR^)^2- 2 * 



(8.6.1) 



We have not yet shown that the Karhunen-Loeve transform is indeed 
the best possible transform for minimizing the overall distortion D tc for a 
given bit allocation. This will be proved under the assumptions that, the 
input variables are Gaussian, that optimal mean squared error quantization 
of the transform coefficients is performed, and that the optimal allocation 
result (8.3.3) (based on the high resolution assumption) holds. 

Before proceeding, we note the following bound on the determinant of 
an autocorrelation matrix that will be useful later. 

Covariance Determinant Bound 

For any autocorrelation (covariance) matrix R„ of a real-valued random 
vector V whose components have zero mean and variances E[V-] = a'K we 
have ' ' : 

k 



detR„ < Yl<r'f. 



i=i 



Furthermore, equality holds if and only if R, is a diagonal matrix. 

As an illustration of this result, consider the case where k = 2. The 
determinant is then given by <t\<t\{\ - p 2 ) where p is the correlation coeffi- 
cient. This clearly satisfies the bound. In this case, and in general, it can be 
shown that the more positively correlated the vector components are, the 
smaller is the determinant. It should also be noted that the determinant 
of an autocorrelation matrix is always nonnegative. 




